Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest Manager] Support both zip & tar archives from Registry #76197

Merged
merged 7 commits into from
Sep 1, 2020

Conversation

jfsiii
Copy link
Contributor

@jfsiii jfsiii commented Aug 27, 2020

Summary

elastic/package-registry#628 will change the archive format from the from .tar.gz to .zip.

Support both formats for greater resiliency/backwards compatibility

  • Restored unzip functions from Remove code for handling .zip files #43764
  • Persist the download value returned by EPR (e.g. /epr/system/system-0.5.3.zip or /epr/system/system-0.5.3.tar.gz) as "archive location" for a package name/version combo.
  • Based on that value, we decide which decompression to use.

Checklist

Testing

Added tests for getBufferExtractor to document tar vs zip behavior

I did a local check reading & installing a few packages while switching the between the master and elastic/package-registry#628.

  • The local registry logs showed the expected .tar.gz vs .zip archives being requested
  • The HTTP responses from installing assets were successful / as expected
  • The appropriate epm-packages saved objects were in place
  • Browsing to the Installed Integrations and Integration detail page worked as expected

Restored unzip functions from elastic#43764

Persist the `download` value returned by EPR (e.g. `/epr/system/system-0.5.3.zip` or  `/epr/system/system-0.5.3.tar.gz`) as "archive key" for a package name/version combo.

The same name&version should return the same archive. The value initially given by the registry.

Based on that value, we decide which decompression to use.
@jfsiii jfsiii added the Team:Fleet Team label for Observability Data Collection Fleet team label Aug 27, 2020
if (!buffer) {
buffer = await fetchArchiveBuffer(pkgName, pkgVersion);
cacheSet(key, buffer);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved the caching from this getOrFetch function to inside the fetchArchiveBuffer

const buffer = await getResponseStream(archiveUrl).then(streamToBuffer);

setArchiveKey(pkgName, pkgVersion, archivePath);
cacheSet(archivePath, buffer);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ensure we always cache a fetched archive buffer

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this cache invalidated if the current package is removed? This is mainly important for package development.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ruflin No, but that's the same behavior as now. I'm not sure if this means it is an issue or if it's being avoided/worked around in some other way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skh @neptunian I thought one of you solve this issue recently or at least debugged it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is #68890 -- it's still open because I misunderstood the original description and tested the wrong things, and then it got moved back in the queue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@skh Thanks for the link. This basically means we can move forward with what we have in this PR and will follow up later in #68890

@jfsiii jfsiii self-assigned this Aug 29, 2020
Copy link
Contributor

@skh skh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM 👍

@ruflin
Copy link
Contributor

ruflin commented Aug 31, 2020

++ on getting this in as it does not seem to change the existing .tar.gz implementation (mostly). Then we can start testing with the zip registry and move over step by step.

@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 31, 2020

@elasticmachine merge upstream

@jfsiii
Copy link
Contributor Author

jfsiii commented Aug 31, 2020

@skh @ruflin what types of tests should I add before merging this?

Or what aspects should I test?

@ruflin
Copy link
Contributor

ruflin commented Sep 1, 2020

@jfsiii I guess for now the most important part is that all things still keep working which seems to be the case. If you could run my registry PR locally to see if it works with your changes, this would be great.

@jfsiii
Copy link
Contributor Author

jfsiii commented Sep 1, 2020

@ruflin sure thing. I'll double check, but I used your branch while developing initially

I only did a quick check of reading & installing a few packages while switching the between the master and elastic/package-registry#628 but 👍

@ruflin
Copy link
Contributor

ruflin commented Sep 1, 2020

@jfsiii Great, then lets get it in! 👍 Can you resolve the conflict?

John Schulz added 2 commits September 1, 2020 13:01
 * Add tests for getBufferExtractor
 * Replace `[aA]rchiveKey*` with `[aA]rchiveLocation*`
@jfsiii jfsiii marked this pull request as ready for review September 1, 2020 17:21
@jfsiii jfsiii requested a review from a team September 1, 2020 17:21
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

@jfsiii jfsiii added release_note:skip Skip the PR/issue when compiling release notes v7.10.0 v8.0.0 labels Sep 1, 2020
@jfsiii
Copy link
Contributor Author

jfsiii commented Sep 1, 2020

@ruflin I fixed the conflict, added some tests and updated the description. I'm planning on merging when CI is 💚

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

async chunks size

id value diff baseline
enterpriseSearch 358.5KB +24.0B 358.5KB

page load bundle size

id value diff baseline
upgradeAssistant 64.8KB +24.0B 64.8KB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@jfsiii jfsiii merged commit 9a7c418 into elastic:master Sep 1, 2020
gmmorris added a commit to gmmorris/kibana that referenced this pull request Sep 2, 2020
* master: (223 commits)
  skip flaky suite (elastic#75724)
  [Reporting] Add functional test for Reports in non-default spaces (elastic#76053)
  [Enterprise Search] Fix various icons in dark mode (elastic#76430)
  skip flaky suite (elastic#76245)
  Add `auto` interval to histogram AggConfig (elastic#76001)
  [Resolver] generator uses setup_node_env (elastic#76422)
  [Ingest Manager] Support both zip & tar archives from Registry (elastic#76197)
  [Ingest Manager] Improve agent vs kibana version checks (elastic#76238)
  Manually building `KueryNode` for Fleet's routes (elastic#75693)
  remove dupe tinymath section (elastic#76093)
  Create APM issue template (elastic#76362)
  Delete unused file. (elastic#76386)
  [SECURITY_SOLUTION][ENDPOINT] Trusted Apps Create API (elastic#76178)
  [Detections Engine] Add Alert actions to the Timeline (elastic#73228)
  [Dashboard First] Library Notification (elastic#76122)
  [Maps] Add mvt support for ES doc sources  (elastic#75698)
  Add setHeaderActionMenu API to AppMountParameters (elastic#75422)
  [ML] Remove "Are you sure" from data frame analytics jobs (elastic#76214)
  [yarn] remove typings-tester, use @ts-expect-error (elastic#76341)
  [Reporting/CSV] Do not fail the job if scroll ID can not be cleared (elastic#76014)
  ...
jfsiii pushed a commit to jfsiii/kibana that referenced this pull request Sep 2, 2020
…ic#76197)

* Quick pass at restoring support for both zip & tar

Restored unzip functions from elastic#43764

Persist the `download` value returned by EPR (e.g. `/epr/system/system-0.5.3.zip` or  `/epr/system/system-0.5.3.tar.gz`) as "archive key" for a package name/version combo.

The same name&version should return the same archive. The value initially given by the registry.

Based on that value, we decide which decompression to use.

* Use template literal vs JSON.stringify for keygen

* Factor unzip/untar logic out to getBufferExtractor

 * Add tests for getBufferExtractor
 * Replace `[aA]rchiveKey*` with `[aA]rchiveLocation*`

* Include given name & version in error message

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
# Conflicts:
#	x-pack/plugins/ingest_manager/server/services/epm/registry/cache.ts
jfsiii pushed a commit that referenced this pull request Sep 2, 2020
… (#76485)

* Quick pass at restoring support for both zip & tar

Restored unzip functions from #43764

Persist the `download` value returned by EPR (e.g. `/epr/system/system-0.5.3.zip` or  `/epr/system/system-0.5.3.tar.gz`) as "archive key" for a package name/version combo.

The same name&version should return the same archive. The value initially given by the registry.

Based on that value, we decide which decompression to use.

* Use template literal vs JSON.stringify for keygen

* Factor unzip/untar logic out to getBufferExtractor

 * Add tests for getBufferExtractor
 * Replace `[aA]rchiveKey*` with `[aA]rchiveLocation*`

* Include given name & version in error message

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
# Conflicts:
#	x-pack/plugins/ingest_manager/server/services/epm/registry/cache.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.10.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants